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2. Abstract 


The Internet Standards Process [1] requires that all IETF Standards 
Track specifications must have "multiple, independent, and 
interoperable implementations" before they can be advanced beyond 
Proposed Standard status. This document specifies the process which 
the IESG will use to determine if a MIB specification document meets 
these requirements. It also discusses the rationale for this 
process. 


3. The Nature of the Problem 


The Internet Standards Process [1] requires that for an IETF 
specification to advance beyond the Proposed Standard level, at least 
two genetically unrelated implementations must be shown to 
interoperate correctly with all features and options. There are two 
distinct reasons for this requirement. 


The first reason is to verify that the text of the specification is 
adequately clear and accurate. This is demonstrated by showing that 
multiple implementation efforts have used the specification to 
achieved interoperable implementations. 
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The second reason is to discourage excessive options and "feature 
creep". This is accomplished by requiring interoperable 
implementation of all features, including options. If an option is 
not included in at least two different interoperable implementations, 
it is safe to assume that it has not been deemed useful and must be 
removed before the specification can advance. 


In the case of a protocol specification which specifies the "bits on 
the wire" exchanged by executing state machines, the notion of 
"interoperability" is reasonably intuitive - the implementations must 
successfully "talk to each other", exchanging "bits on the wire", 
while exercising all features and options. 


In the case of an SNMP Management Information Base (MIB) 
specification, exactly what constitutes "interoperation" is less 
obvious. This document specifies how the IESG has decided to judge 
"MIB specification interoperability" in the context of the IETF 
Standards Process. 


There are a number of plausible interpretations of MIB specification 
interoperability, many of which have merit but which have very 
different costs and difficulties in realization. 


The aim is to ensure that the dual goals of specification clarity and 
feature evaluation have been met using an interpretation of the 
concept of MIB specification interoperability that strikes a balance 
between testing complexity and practicality. 


4. On The Nature of MIB specifications 


Compared to "state machine" protocols which focus on procedural 
specifications, a MIB specification is much more data oriented. To 
over-generalize, in a typical MIB specification the collection of 
data type and instance specifications outnumbers inter-object 
procedural or causal semantics by a significant amount. 


A central issue is that a MIB specification does not stand alone; it 
forms the access interface to the instrumentation underneath it. 
Without the instrumentation, a MIB has form but no values. Coupled 
with the large number of objects even in a simple MIB specification, 
a MIB specification tends to have more of the look and feel of an API 
or a dictionary than a state machine protocol. 


It is important to distinguish between assessing the interoperability 


of applications which may use or interact with MIBs, and the MIBs 
themselves. It is fairly obvious that "black-box testing" can be 


O’Dell, et. al. Best Current Practice [Page 2] 


RFC 2438 Advancement of MIB specifications October 1998 


applied to such applications and that the approach enjoys a certain 
maturity in the software engineering arts. A MIB specification, on 
the other hand is not readily amenable to black box test plans. 


5. Discussion and Recommended Process 


In order to meet their obligations under the IETF Standards Process, 
the Operations and Management Area Directors and the IESG must be 
convinced that each MIB specification advanced to Draft Standard or 
Internet Standard status is clearly written, that there are the 
required multiple interoperable implementations, and that all options 
have been implemented. There are multiple ways to achieve this goal. 
Appendix A lists some testing approaches that could be used when 
attempting to document multiple implementations. 


The Full Coverage or Stimulus-Response approaches are very through, 
and would increase confidence that the requirement has been met, if 
applied. However, experience in real-world software engineering 
makes it clear that such confidence comes at an extremely high price; 
even with the most exhaustive testing, it is often not clear what 
precisely has been demonstrated by such testing. We believe that 
both of those standards of evidence are materially beyond what can be 
reasonably accomplished in an operational sense, and achieving the 
requisite semantic specifications are even more unlikely. 


Therefore, the Operations and Management Area and the IESG have 
adopted a more pragmatic approach to determining the suitability of a 
MIB specification for advancement on the standards track beyond 
Proposed Standard status. Each MIB specification suggested for 
advancement must have one or more advocates who can make a convincing 
argument that the MIB specification meets the multiple implementation 
and feature support requirements of the IETF Standards Process. The 
specific way to make the argument is left to the advocate, but will 
normally include reports that basic object comparison testing has 
been done. 


Thus any recommendation for the advancement of a MIB specification 
must be accompanied by an implementation report, as is the case with 
all requests for the advancement of IETF specifications. The 
implementation report must include the reasons why the IESG should 
believe that there are multiple implementations of the MIB 
specification in question and that the all of the MIB objects in the 
specification to be advanced are supported in more than one 
implementation. But note that the prime concern of the IESG will be 
that the underlying reasons for the interoperable implementations are 
met, i.e., that the text of the specification is clear and 
unambiguous, and that features of the specification which have not 
garnered support have been removed. 
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The implementation report will be placed on the IETF web page along 
with the other pre-advancement implementation reports and will be 
specifically referred to in the IETF Last-Call. As with all such 
implementation reports, the determination of adequacy is made by the 
Area Director(s) of the relevant IETF Area. This determination of 
adequacy can be challenged during the Last-—Call period. 


6. Security Considerations 


Some may view this policy as possibly leading to a reduction in the 
level of confidence people can have in MIB specifications but the O&M 
Area Directors and the IESG feel that it will adequately ensure a 
reasonable evaluation of the level of clarity of MIB specifications 
and to ensure that unused options can be identified and removed 
before the advancement of a specification. 


Good, clearly written MIB specifications can be of great assistance 
in the management of the Internet and other networks and thus assist 
in the reduction of some types of security threats. 
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Appendix A 


A. 


Some Testing Alternatives 


The IESG debated a number of interoperability and testing models in 
formulating this specification. The following list is not an 
exhaustive enumeration of the alternatives, but it does capture the 
major plausible models which were examined in the course of the 
discussion. 


Basic Object Comparison 


Assume the requisite two genetically unrelated implementations of the 
MIB in an SNMP agent and an SNMP management station which can do a 
"MIB Dump" (extract the complete set of MIB object types and values 
from the agent implementation). Extract a MIB Dump from each 
implementation and compare the two dumps to verify that both provide 
the complete set of mandatory and optional objects and that the 
individual objects are of the correct types. 


A.2 Stimulus/Response Testing 


Proceed as in A.1, but in addition, comprehensively exercise the two 
(network) elements containing the agent implementations to verify 
that all the MIB objects reflect plausible values in operational 
conditions. An even stricter interpretation would require that the 
MIB objects in the two network elements track identically given the 
identical stimulus. While this would test "read-only" or 
"monitoring" information obtained from the underlying 
instrumentation, it is important to observe that such instrumentation 
is actually an *application* which uses the MIB and is not part of 
the MIB itself. 


A.3 Full Coverage Testing 


This model extends the notion of Stimulus/Response Testing to its 
logical extreme. The MIB is viewed as an API and the software 
engineering notion of full coverage testing is applied to a MIB. 

This involves exercising all paths through the causal semantics and 
verifying that all objects change state correctly in all cases. 
Again, note that much more than the MIB definition is being exercised 
and evaluated. 
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Full Copyright Statement 
Copyright (C) The Internet Society (1998). All Rights Reserved. 


This document and translations of it may be copied and furnished to 
others, and derivative works that comment on or otherwise explain it 
or assist in its implementation may be prepared, copied, published 
and distributed, in whole or in part, without restriction of any 
kind, provided that the above copyright notice and this paragraph are 
included on all such copies and derivative works. However, this 
document itself may not be modified in any way, such as by removing 
the copyright notice or references to the Internet Society or other 
Internet organizations, except as needed for the purpose of 
developing Internet standards in which case the procedures for 
copyrights defined in the Internet Standards process must be 
followed, or as required to translate it into languages other than 
English. 


The limited permissions granted above are perpetual and will not be 
revoked by the Internet Society or its successors or assigns. 


This document and the information contained herein is provided on an 
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 
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